Repository navigation
test: measure first-attempt workflow conformance - #142
Merged
trvon merged 1 commit intoOct 6, 2026
Merged
Conversation
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Two moderate evaluator issues remain around blank API metadata and first-attempt batch detection.
Review effort: Lite
Findings: None
What changed in this PR
Adds first-attempt workflow-conformance measurements, deterministic evaluator tests, metadata collection, and documentation while preserving existing routing behavior.
Changes:
- Separates first-attempt validity from semantic correctness.
- Adds workflow, evaluator, and cadence/rework tests.
- Expands MJS discovery/linting and documents measurement limitations.
| File | Summary |
|---|---|
vitest.config.ts |
Discovers TypeScript and MJS tests. |
test/workflow-guidance.test.ts |
Validates workflow examples and cadence semantics. |
test/e2e/controller-routing-evaluation.mjs |
Provides evaluation and metadata helpers; changes requested for blank API handling and first-batch detection. |
test/e2e/controller-routing-conformance.mjs |
Integrates evaluation and runtime metadata reporting. |
test/controller-routing-evaluation.test.mjs |
Adds deterministic evaluator and non-mutation tests. |
docs/CONTROLLER_ROUTING_EVAL.md |
Documents measurement boundaries and baseline limitations. |
biome.json |
Enables linting for new MJS files. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
Depends on #140; targets
feat/lifecycle-inspection. Review after the testing-contract and lifecycle layers. No merge or auto-merge requested.Why this layer
The historical routing evaluation documented repaired workflow-schema attempts despite final routing success. The current unchanged-copy baseline did not reproduce that problem: one requested model/configuration, six existing fixtures, all four checklist items passing and zero repairs. Compact tool copy, validator, and scenario prompts remain unchanged; this PR does not claim a guidance fix or model-performance improvement.
Changes
firstAttemptValidandfirstAttemptSemanticseparately from the existing four routing judgments. Repaired first calls stay first-pass failures; accepted-but-wrong arguments remain distinct from schema validity.